Skip to content

Conversation

@edsiper
Copy link
Member

@edsiper edsiper commented Sep 1, 2025

Fixes #10793

  • Added logic to use the current host time when both timeUnixNano and observedTimeUnixNano are absent, removing the previous hard requirement for a timestamp in OTEL log records.

  • Updated to prioritize observed_time_unix_nano when time_unix_nano is missing and to omit observed_timestamp metadata when no value is available.


Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

  • New Features
    • Logs now accept observed time as a valid timestamp source when primary timestamps are absent.
  • Bug Fixes
    • Missing timestamps no longer cause errors; events now fall back to the current time.
    • Observed timestamp is emitted only when present, avoiding empty or zero values.
  • Documentation
    • Updated examples to reflect new handling for missing timestamps and revised error expectations.
  • Tests
    • Adjusted fixtures to verify fallback-to-current-time behavior.
    • Removed a redundant case related to missing timestamps.

Added logic to use the current host time when both timeUnixNano and observedTimeUnixNano
are absent, removing the previous hard requirement for a timestamp in OTEL log records.

Updated to prioritize observed_time_unix_nano when time_unix_nano is missing and to omit
observed_timestamp metadata when no value is available.

Signed-off-by: Eduardo Silva <[email protected]>
@edsiper edsiper added this to the Fluent Bit v4.1 milestone Sep 1, 2025
@coderabbitai
Copy link

coderabbitai bot commented Sep 1, 2025

Walkthrough

Adds observed_time_unix_nano as a valid timestamp source for OTEL logs, falls back to the host current time when both timestamp fields are missing, and emits observed_timestamp metadata only when non-zero. Tests and README updated to reflect acceptance of records without timestamps.

Changes

Cohort / File(s) Summary
OTel logs timestamp handling (C sources)
plugins/in_opentelemetry/opentelemetry_logs.c, src/opentelemetry/flb_opentelemetry_logs.c
Add #include <fluent-bit/flb_time.h>; resolve timestamps in order: time_unix_nanoobserved_time_unix_nano → current time via flb_time_get; emit observed_timestamp only when non-zero; replace error-on-missing-timestamp with current-time fallback.
Tests and fixtures
tests/internal/data/opentelemetry/README.md, tests/internal/data/opentelemetry/test_cases.json
Update README error example code; change missing_timestamp test to expect successful parsing (timestamp defaulted) and remove multiple_records_no_timestamp_after_valid_ones test case.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor Client as OTLP Sender
  participant Parser as OTEL Logs Parser
  participant Encoder as Log Event Encoder
  participant Clock as System Clock

  Client->>Parser: Send LogRecord (time_unix_nano?, observed_time_unix_nano?)
  alt time_unix_nano > 0
    Parser->>Encoder: set timestamp = time_unix_nano
  else observed_time_unix_nano > 0
    Parser->>Encoder: set timestamp = observed_time_unix_nano
  else no timestamps
    Parser->>Clock: flb_time_get()
    Clock-->>Parser: current time
    Parser->>Encoder: set timestamp = current time
  end

  note right of Encoder: observed_timestamp included only if observed_time_unix_nano != 0
  Encoder-->>Client: Encoded log event
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Assessment against linked issues

Objective Addressed Explanation
Accept OTEL log records without timestamp and default to current host time (#10793)
Treat observedTimestamp as optional and usable when present (#10793)
Avoid returning FLB_OTEL_LOGS_ERR_MISSING_TIMESTAMP for records without timestamps (#10793)

Suggested reviewers

  • koleini
  • fujimotos

Poem

I hop through timestamps, quick and keen,
From nano ticks to times unseen—
If none are found, I fetch “now” with grace,
I stamp each log and keep its place.
Rabbit packs events, tidy and keen. 🐇🕰️


📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 341ba03 and 4387ff0.

📒 Files selected for processing (2)
  • tests/internal/data/opentelemetry/README.md (1 hunks)
  • tests/internal/data/opentelemetry/test_cases.json (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
  • tests/internal/data/opentelemetry/README.md
  • tests/internal/data/opentelemetry/test_cases.json
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (29)
  • GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
  • GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
  • GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_ARROW=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
  • GitHub Check: PR - fuzzing test
  • GitHub Check: pr-compile-centos-7
  • GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-22.04, clang-12)
  • GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-22.04, clang-12)
  • GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-24.04, clang-14)
  • GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
✨ Finishing Touches
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch otel-logs-timestamp

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

  • Add @coderabbitai ignore or @coderabbit ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

  • Visit our Status Page to check the current availability of CodeRabbit.
  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
src/opentelemetry/flb_opentelemetry_logs.c (2)

210-218: Bug: wrong error code returned for invalid spanId hex character.

On non-hex spanId chars you return UNEXPECTED_TIMESTAMP_TYPE instead of INVALID_SPAN_ID.

-                return -FLB_OTEL_LOGS_ERR_UNEXPECTED_TIMESTAMP_TYPE;
+                return -FLB_OTEL_LOGS_ERR_INVALID_SPAN_ID;

119-120: Fix tv_nsec computation in flb_time_from_uint64
The current implementation subtracts seconds from the full nanosecond count—yielding tv_nsec values far exceeding 1e9 (e.g., 1640995198359004800 for input 1640995200000000000). Replace with modulo to correctly compute sub‐second nanoseconds:

--- a/include/fluent-bit/flb_time.h
+++ b/include/fluent-bit/flb_time.h
@@ static inline void flb_time_from_uint64(struct flb_time *dst, uint64_t value)
-    dst->tm.tv_sec = (long) (value / 1000000000L);
-    dst->tm.tv_nsec = (long) (value - dst->tm.tv_sec);
+    dst->tm.tv_sec  = (long) (value / 1000000000ULL);
+    dst->tm.tv_nsec = (long) (value % 1000000000ULL);
🧹 Nitpick comments (4)
tests/internal/data/opentelemetry/test_cases.json (1)

40-46: Align “missing_timestamp” expectation with current JSON group-body shape.

JSON path doesn’t emit an empty scope when no scope object is present, while other tests (e.g., “valid_basic_log”) also omit it. Recommend removing the empty scope here for consistency.

   "missing_timestamp": {
     "input": {"resourceLogs": [{"scopeLogs": [{"logRecords": [{"body": {"stringValue": "test"}}]}]}]},
     "expected": {
       "group_metadata": {"schema":"otlp","resource_id":0,"scope_id":0},
-      "group_body": {"resource":{},"scope":{}},
+      "group_body": {"resource":{}},
       "log_metadata": {"otlp":{}},
       "log_body": {"log": "test"}
     }
   },
src/opentelemetry/flb_opentelemetry_logs.c (3)

248-274: Emit observed_timestamp when observedTimeUnixNano is an integer too.

Currently metadata emission only handles string type. Accept positive integers as well (spec-compliant and consistent with time resolution above).

-    if (observed_time_unix_nano != NULL && observed_time_unix_nano->type == MSGPACK_OBJECT_STR) {
+    if (observed_time_unix_nano != NULL) {
+        if (observed_time_unix_nano->type == MSGPACK_OBJECT_STR) {
             memset(timestamp_str, 0, sizeof(timestamp_str));
             if (observed_time_unix_nano->via.str.size < sizeof(timestamp_str)) {
                 strncpy(timestamp_str,
                         observed_time_unix_nano->via.str.ptr,
                         observed_time_unix_nano->via.str.size);
             }
             else {
                 strncpy(timestamp_str,
                         observed_time_unix_nano->via.str.ptr,
                         sizeof(timestamp_str) - 1);
             }
-        for (i = 0; i < strlen(timestamp_str); i++) {
+        size_t ts_len = strlen(timestamp_str);
+        for (i = 0; i < ts_len; i++) {
             if (!isdigit((unsigned char) timestamp_str[i])) {
                 timestamp_str[0] = '\0';
                 break;
             }
         }
 
         if (strlen(timestamp_str) > 0) {
             timestamp_uint64 = strtoull(timestamp_str, NULL, 10);
 
             flb_log_event_encoder_append_metadata_values(encoder,
                                                         FLB_LOG_EVENT_STRING_VALUE("observed_timestamp", 18),
                                                         FLB_LOG_EVENT_INT64_VALUE(timestamp_uint64));
         }
-    }
+        }
+        else if (observed_time_unix_nano->type == MSGPACK_OBJECT_POSITIVE_INTEGER) {
+            flb_log_event_encoder_append_metadata_values(encoder,
+                                                         FLB_LOG_EVENT_STRING_VALUE("observed_timestamp", 18),
+                                                         FLB_LOG_EVENT_INT64_VALUE(observed_time_unix_nano->via.u64));
+        }
+    }

100-111: Micro: avoid repeated strlen() in loops.

Cache the length once to keep O(n).

-            for (i = 0; i < strlen(timestamp_str); i++) {
+            size_t ts_len = strlen(timestamp_str);
+            for (i = 0; i < ts_len; i++) {
-        for (i = 0; i < strlen(timestamp_str); i++) {
+        size_t ts_len = strlen(timestamp_str);
+        for (i = 0; i < ts_len; i++) {

Also applies to: 260-266


242-244: Use the function parameters for metadata key instead of hardcoding.

Preserves flexibility and aligns with the function signature.

-    flb_log_event_encoder_append_metadata_values(encoder,
-                                                 FLB_LOG_EVENT_CSTRING_VALUE(FLB_OTEL_LOGS_METADATA_KEY));
+    flb_log_event_encoder_append_metadata_values(
+        encoder,
+        FLB_LOG_EVENT_STRING_VALUE(logs_metadata_key, logs_metadata_key_len));
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 4db4f21 and 341ba03.

📒 Files selected for processing (4)
  • plugins/in_opentelemetry/opentelemetry_logs.c (3 hunks)
  • src/opentelemetry/flb_opentelemetry_logs.c (2 hunks)
  • tests/internal/data/opentelemetry/README.md (1 hunks)
  • tests/internal/data/opentelemetry/test_cases.json (1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
plugins/in_opentelemetry/opentelemetry_logs.c (3)
src/flb_mp.c (1)
  • flb_mp_map_header_append (345-349)
include/fluent-bit/flb_time.h (1)
  • flb_time_from_uint64 (87-91)
src/flb_log_event_encoder.c (1)
  • flb_log_event_encoder_set_timestamp (276-287)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (29)
  • GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
  • GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
  • GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
  • GitHub Check: PR - fuzzing test
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_ARROW=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
  • GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
  • GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
  • GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-22.04, clang-12)
  • GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-24.04, clang-14)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
  • GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-22.04, clang-12)
  • GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
  • GitHub Check: pr-compile-centos-7
🔇 Additional comments (6)
tests/internal/data/opentelemetry/README.md (1)

139-140: Doc update matches new behavior (no hard requirement on timestamp).

The error code example change looks correct.

src/opentelemetry/flb_opentelemetry_logs.c (2)

24-25: Correct include.

Including flb_time.h is needed for flb_time_get usage.


77-81: Good: fallback to host time when both timestamps are absent.

Matches PR intent and prevents unnecessary drops.

plugins/in_opentelemetry/opentelemetry_logs.c (3)

24-25: Correct include.

Needed for explicit flb_time_get usage.


221-226: Observed timestamp metadata is now conditional.

Good: emits only when present (non-zero).


520-527: Timestamp resolution order is correct (time → observed_time → now).

Matches OTLP intent and PR goal.

@edsiper edsiper force-pushed the otel-logs-timestamp branch from 341ba03 to 4387ff0 Compare September 1, 2025 18:22
@edsiper edsiper merged commit a46116d into master Sep 1, 2025
44 checks passed
@edsiper edsiper deleted the otel-logs-timestamp branch September 1, 2025 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Regression: Fluent-bit now requires timestamps for OTEL Logs

2 participants